Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(datasource): add debian datasource #30071

Merged
merged 81 commits into from
Aug 28, 2024
Merged

feat(datasource): add debian datasource #30071

merged 81 commits into from
Aug 28, 2024
+2,697 −0

Conversation

oxdev03
Copy link
Contributor

@oxdev03 oxdev03 commented Jul 7, 2024

Changes

  • Built upon @Ka0o0 implementation of the Debian datasource.
  • Includes minor improvements.
  • Addresses coverage issues identified in the previous review of the Debian datasource.
  • Fixes earlier review findings (except the caching issue).

Context

Resolves: #7041, #24906

Concept

Based on the discussion and the reused implementation (see the related issue for more details):

  1. Evaluation of Provided or Default Registry URL

    • Example: The URL https://ftp.debian.org/debian?release=bullseye&components=main,contrib&binaryArch=amd64 is evaluated to:
      • .../debian/dists/bullseye/main/binary-amd64
      • .../debian/dists/bullseye/contrib/binary-amd64
    • Supports multiple component URLs.

  2. Check for Existing Compressed Package

    • If the compressed Packages.gz file has already been downloaded and extracted:
      • Send a HEAD request to check if the upstream file has been updated.
    • If not already downloaded:
      • Download the compressed file.
      • Extract the contents.
      • Delete the compressed file.
    • Currently, only gzip compression is supported.

  3. Iterate through Package Index

    • Gather relevant information for the requested package.
    • Stop iteration, if found

  4. Return relevant release information

Documentation (please check one with an [x])

  • I have updated the documentation, or
  • No documentation update is required

included documentation for new datasource

How I've tested my work (please select one)

I have verified these changes via:

  • Code inspection only, or
  • Newly added/modified unit tests, or
  • No unit tests but ran on a real repository, or
  • Both unit tests + ran on a real repository

Real Test

Open Points

  • Does the current caching implementation for the datasource need to be reworked? Can the Renovate team provide a concept/details for this? (e.g., invalidation of the compressed file based on its timestamp)

  • Is the current approach generic enough? Are there any points that were not considered?

  • If significant rework is required, please let me know in advance.

@CLAassistant
Copy link

CLAassistant commented Jul 7, 2024 •
edited
Loading

CLA assistant check
All committers have signed the CLA.

@viceice viceice self-requested a review July 7, 2024 17:15
secustor
secustor requested changes Jul 7, 2024
View reviewed changes
Copy link
Collaborator

@secustor secustor left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your contribution, and the well documented functionality!

This is my first pass as there is a lot going on here and we have to be sure that the caching is setup efficiently.

Would this work for every APT repository?

lib/modules/datasource/deb/index.ts Outdated Show resolved Hide resolved
lib/modules/datasource/deb/index.ts Outdated Show resolved Hide resolved
lib/modules/datasource/deb/index.ts Outdated Show resolved Hide resolved
lib/modules/datasource/deb/index.ts Outdated Show resolved Hide resolved
lib/util/fs/index.ts Outdated Show resolved Hide resolved
lib/modules/datasource/deb/index.ts Outdated Show resolved Hide resolved
lib/modules/datasource/deb/index.ts Outdated Show resolved Hide resolved
@secustor
Copy link
Collaborator

secustor commented Jul 7, 2024

Does the current caching implementation for the datasource need to be reworked? Can the Renovate team provide a concept/details for this? (e.g., invalidation of the compressed file based on its timestamp)

We should start simple and then start to optimize.
The Package.gz will be cached non transparently by the http layer, so the file will be downloaded only when necessary.
I suggest to simply cache the extracted file on a time base ( extract it to a different function and use the annotation ).

@oxdev03
Copy link
Contributor Author

oxdev03 commented Jul 7, 2024 •
edited
Loading

This is my first pass as there is a lot going on here and we have to be sure that the caching is setup efficiently.

Thx for the initial review, I will fix the findings soon.

Would this work for every APT repository?

should work with every apt repository, which follows the https://wiki.debian.org/DebianRepository standard.

viceice
viceice reviewed Jul 9, 2024
View reviewed changes
lib/modules/datasource/deb/index.spec.ts Outdated Show resolved Hide resolved
lib/modules/datasource/deb/readme.md Outdated Show resolved Hide resolved
lib/modules/datasource/repology/readme.md Outdated Show resolved Hide resolved
@rarkins rarkins requested review from viceice and secustor July 20, 2024 11:12
secustor
secustor requested changes Jul 31, 2024
View reviewed changes
lib/modules/datasource/deb/index.spec.ts Outdated Show resolved Hide resolved
lib/modules/datasource/deb/index.ts Outdated Show resolved Hide resolved
lib/modules/datasource/deb/index.ts Show resolved Hide resolved
lib/modules/datasource/deb/index.ts Outdated Show resolved Hide resolved
viceice
viceice reviewed Aug 5, 2024
View reviewed changes
lib/modules/datasource/deb/index.ts Outdated Show resolved Hide resolved
lib/modules/datasource/deb/index.ts Outdated Show resolved Hide resolved
lib/modules/datasource/deb/index.ts Show resolved Hide resolved
lib/modules/datasource/deb/index.ts Outdated Show resolved Hide resolved
lib/modules/datasource/deb/readme.md Outdated Show resolved Hide resolved
lib/modules/datasource/deb/index.ts Outdated Show resolved Hide resolved
lib/util/fs/index.ts Outdated Show resolved Hide resolved
lib/modules/datasource/deb/url.ts Show resolved Hide resolved
oxdev03 and others added 14 commits August 22, 2024 18:43
@oxdev03
move vars to describe block
97018b1
@oxdev03
cleanup cache dir
bb9f352
@oxdev03
validate http head call using nock options
5698896
@zharinov @oxdev03
test(regex-manager): Separate arrange/act/assert with spaces (renovat…
ee03bdf 
…ebot#30972)
@renovate @oxdev03
fix(deps): update ghcr.io/renovatebot/base-image docker tag to v7.16.1 (
c3ea33a 
renovatebot#30974)

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
@amezin @oxdev03
fix(manager/git-submodules): get recorded submodule commit (renovateb…
2923d8a 
…ot#30976)
@renovate @oxdev03
chore(deps): update dependency mkdocs-material to v9.5.33 (renovatebo…
62d80b1 
…t#30981)

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
@renovate @oxdev03
feat(deps): update ghcr.io/renovatebot/base-image docker tag to v7.17…
e914d9a 
….0 (renovatebot#30983)

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
@renovate @oxdev03
chore(deps): update github/codeql-action action to v3.26.5 (renovateb…
5f648f3 
…ot#30986)

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
@renovate @oxdev03
chore(deps): update dependency type-fest to v4.25.0 (renovatebot#30987)
80f51c5 
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
@jazzlyn @oxdev03
docs(docker): add ghworkflows example for GAR with Workload Identity (r…
2bd4ede 
…enovatebot#30692)
@renovate @oxdev03
chore(deps): update dependency google-github-actions/auth to v2.1.5 (r…
a1a01a1 
…enovatebot#30994)

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
@renovate @oxdev03
chore(deps): update dependency renovatebot/github-action to v40.2.6 (r…
2cb37e9 
…enovatebot#30995)

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
@oxdev03
Merge branch 'main' into feat/debian-datasource
78399ec
@oxdev03 oxdev03 requested a review from viceice August 24, 2024 09:04
@rarkins rarkins requested a review from secustor August 26, 2024 09:58
viceice
viceice reviewed Aug 26, 2024
View reviewed changes
lib/modules/datasource/deb/index.ts Outdated Show resolved Hide resolved
lib/modules/datasource/deb/index.ts Outdated Show resolved Hide resolved
lib/util/cache/package/types.ts Outdated Show resolved Hide resolved
lib/modules/datasource/deb/index.ts Outdated Show resolved Hide resolved
@rarkins rarkins requested a review from viceice August 27, 2024 03:49
@rarkins rarkins enabled auto-merge August 27, 2024 07:48
@rarkins rarkins added this pull request to the merge queue Aug 28, 2024
Merged via the queue into renovatebot:main with commit c3958c9 Aug 28, 2024
38 checks passed
@secustor
Copy link
Collaborator

@oxdev03 Thanks for the contribution and pushing this through! 🚀

@renovate-release
Copy link
Collaborator

🎉 This issue has been resolved in version 38.57.0 🎉

The release is available on:

Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Finkregh Finkregh Finkregh left review comments

@zharinov zharinov zharinov left review comments

@secustor secustor secustor approved these changes

@viceice viceice Awaiting requested review from viceice

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
10 participants
@oxdev03 @CLAassistant @secustor @renovate-release @Finkregh @zharinov @viceice @rarkins @amezin @jazzlyn